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While the agenda of the 1960 's was one of 
at maximum feasible speedy the by-words of the 1970«s are 
accountability and its correlates: effectiveness^ efficiency, 
quality. In statewide university systems accountability is a 
formidable task. A major difficulty comes from the diversity of goals 
and disagreement or confusion over measuring the achievement of those 
goals. Two major aspects of the problem are contradictory perceptions 
of "who is accountable for what" and actual operation of a system of 
accountability. There has been a failure in measuring performance and 
to distinguish and relate input variables, process variables, and 
output variables. Many previous evaluations by accrediting agencies 
and external reviewers have tended to focus of the input variables of 
quality and reputation of faculty, quality of students, and adequacy 
of resources. This orientation overlooks the impact of a program upon 
the ultimate success of its students and, of equal importance, it 
tends to neglect the manner in which teachers and learners interact 
for some purpose. By delineating and clarifying the evaluation 
process through the use of a specific checklist confusion can by 
minimized and areas of agreement can be enlarged. (Author/KE) 
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Following an unprecedented period of seemingly 
limitless, irreversible growth, the mood of American 
higher education has shifted to one of self-reflection 
and reassessment. While the agenda of the 60's was 
one of expansion at maximum feasible speed, the 
bywords of the 70's are accountability and its cor- 
relates:- effectiveness, efficiency and quality. 

In a statewide university system, accountability 
is a formidable task. A major difficulty comes from 
the diversity of goals and disagreement or confusion 
over measuring the achievement of those goals. For 
example, the accountability process within the 
SUNY system involves 72 campuses, university Cen- 
tral Administration, the New York State Division of 
Budget, the State Education Department, and the 
j<\ legislature. To expect agreement would be Utopian; 
— to work for enlarging areas of agreement and mini- 
mizing confusion and conflict is not. 

It is from this perspective that the Comparative 
Development Studies Center (CDSC) approached the 
task of studying the SUNY process of accountability 
and evaluating efficiency, effectiveness and quality. 
Im In addition to an analysis of prevailing concepts 
from relevant literature, the Center's objective was 
\ to identify suitable criteria and recommendations 
from participants in the process. This was done 
chief'y through the use of questionnaires and inten- 
sive interviews with 43 officials from all levels of 
university governance. 
O 
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This COMMENT features two major aspects of the 
study: section one presents an analysis of the con- 
tradictory perceptions of "who is accountable for 
what" in New York higher education, discusses var- 
ious interinstitutional accountability roles, and 
highlights some of the recommendations designed 
to mediate among opposing perceptions and roles. 
Section two explores the problems ot actually oper- 
ating a system of accountability through the evalua- 
tion of efficiency, effectiveness and quality, and of- 
fers a checklist of evaluation criteria. The checklist 
is an attempt to clarify and relate the variables of 
performance evaluation. 



ACCOUNTABILITY AUTHORITY ROLES 



Accountability is a protean word. For purposes of 
this study it was broadly defined as: those prac- 
tices, policies and procedures by which a public 
higher educational system demonstrates its legit- 
imacy and value as a state-supported enterprise. It 
is the system by which a public university indicates 
to the citizens and their elected and nonelected rep- 
resentatives the appropriateness of its goals, the ef- 
fectiveness and quality of its programs, and the ef- 
ficiency of its operations in providing service to the 
state. 

In New York accountability is characterized by nu- 
merous levels of control and review and a wide de- 
gree of confusion and conflict over appropriate res- 
ponsibilities. For example, many of the individuals 
contacted in the study complained about the extent 
of detailed fiscal controls exercised by the Division, 
of Budget (DOB). They were also critical about the 
impact of the State Education Department's policies 
on the individual campuses of the SUNY system. 
Officials from DOB and the State Education Depart- 
ment, in turn, defended their policies and argued 
that their impact on SUNY would be lessened if 
SUNY, and SUNY Central Administration in particu- 
lar, would exercise more effective control over the 
system. 

SUNY Central Administration occupies a particu- 
larly critical and difficult position in the account- 
ability system. On the one hand, it is expected to 



act as an agent of state government, clarifying and 
implementing state policies and priorities. The 
campuses, on the other hand, believe that Central's 
role should be that of a spokesman for the cam- 
puses, communicating campus needs to state 
authorities, protecting the campus from direct state 
intervention, and acting as a mediator between the 
campus and the state. In practice, Central Admini- 
stration attempts to fulfill both sets of conflicting 
expectations, but Is not seen as performing either 
role. This tension between accountability to the 
state and responsiveness to campus needs may be 
characteristic of the central administration of any 
multicampus university system. 

The individuals interviewed from Central Admini- 
stration indicated that SUNY defines itself as a 
single university with multiple campuses rather than 
a confederation of semi-autonomous universities. 
V^his definition is based on the belief that the SUNY 
Board of Trustees and Central Administration exer- 
cise centralization for policy development but allow 
campus autonomy for policy implementation and 
day-to-day administration. Much of the evidence 
gathered in our study would tend to question the 
accuracy of this conception. For example. Central 
administrators were the only participants to refer to 
this division of centralized policy and decentralized 
implementation. Comments and criticisms from the 
campuses, as well as from agencies external to 
SUNY, suggest that there is not sufficient decentral- 
ization and flexibility in the area of campus opera- 
tions and policy implementation. Many specifically 
mentioned inadequate leadership from Central ad- 
ministrators in developing broad policy and setting 
parameters within which the campuses may operate 
with relative autonomy. In fact, several Indicated 
that Central Administration's accountability to the 
state should begin with defining the objectives and 
missions of the system and clarifying SUNYwide 
policies and those of the external agencies. More- 
over, campus respondents often described Central's 
performance as that of a line agency rather than a 
central coordinating staff and criticized its conflict- 
ing policies and lack of decisiveness. 

Some respondents from Central Administration 
defined their role as that of campus advocate-nego- 
tiating budget requests with DOB and the legisla- 
ture, providing political antennae for the campuses 
, (keeping campuses informed of developments in the 
legislature and DOB), and fighting for program ap- 
proval. They also viewed Central's role as the pro- 
cessors and providers of data relevant to campus 
management needs. The campuses, however, indi- 
cated that Central should become more active in pro- 
tecting campuses from excessive fiscal entangle- 
ment with DOB and from perceived threats to auto- 
nomy by the State Education Department. They 
were also critical of the lack of adequate information 
sharing, its usefulness, and the extent to which 
Central Administration informed them of the ulti- 
mate use of data requested. 



In addition, respondents from the external review 
agencies had another conception of Central Admini- 
stration. They criticized Central for being too sen- 
sitive to campus autonomy, for being protective of 
the campuses, for failing to make "hard-nosed" 
decisions regarding program elimination and the 
denial of campus program requests, and for gen- 
erally following a bottom-up rather than top-down 
management philosophy. 

Given these conflicting demands and contradic- 
tory assessments of Central Administration, it 
would appear that Central cannot satisfy the expec- 
tations of all decision-making levels. Several res- 
pondents pointed to the surprising degree of hos- 
tility and suspicion between the campuses and Cen- 
tral Administration and the external review agencies. 

Some element of conflict between Central Admini- 
stration and the campuses, and Central Administra- 
tion and external authorities is inevitable. Conflict 
is endemic to a statewide educationa^ system since 
the various interests (students, faculty, administra- 
tors, legislators, the Regents, and special interest 
groups) have a stake in the decision-making pro- 
cess, and they all approach the same issues from 
differing institutional needs and perspectives. More- 
over they have differing conceptions of what ac- 
countability to the state should mean. For example, 
in response to a query for specific ways the cam- 
puses should be held accountable to the state, we 
noted significant differences in perceptions between 
the campus respondents vis-a-vis the respondents 
from Central Administration and the external review 
agencies. Campus respondents tended to define 
accountability in terms of programs, services, or 
activities; while Central and external respondents 
defined it in terms of processes. In the area of fiscal 
accountability, for instance, campus respondents 
defined accountability by reference to broad state- 
ments about program quality. Respondents from 
Central Administration, DOB and the legislature, on 
the other hand, tended to define accountability in 
terms of the budget process. While campus person- 
nel defined accountability abstractly, other respon- 
dents made frequent references to specific respon- 
sibilities, reporting relationships, and mechanisms 
of coordination. Noncampus respondents often 
cited the state's master planning process as a defin- 
ing characteristic of accountability. This process 
was not mentioned by any campus respondents, 
which may give indirect evidence as to why the edu- 
cational planning process has not been very effec- 
tive in New York state. 

In addition, there was a tendency to place the res- 
ponsibility for weaknesses in the accountability pro- 
cess at a level different from the respondent's. In 
the area of establishing institutional goals, for 
example. Central Administration criticized the cam- 
puses for not being more explicit in goal articulation 
as campus planning documents seldom go beyond 
, broad statements of purpose. Campus respondents. 



as stated earlier, criticized Central Administration 
for not clearly setting the overall mission of the sys- 
tem and providing parameters within which the cam- 
pus can plan; and, in turn, some respondents from 
Central Administration, as well as the campuses, 
called upon the state to set more clearly the para- 
meters for public higher education in New York. 

RECOMMENDATIONS 

In order to mediate among these conflicting ex- 
pectations and assessments of the SUNY govern- 
ance system and more clearly delineate responsibi- 
lities and demonstrate accountability, several 
changes in policy, systemization and procedure are 
required. Some of the study's policy recommenda- 
tions for effecting such changes are outlined below. 

In terms of the relationship between SUNY Central 
Administration and the campuses it was recom- 
mended that: 

• Central Administration articulate a specific mission state- 
ment which spells out the operational Implications of its phil- 
osophy of centralized policy with decentralized Implementation. 

• Central Administration develop "Central Administration 
internships" for campus administrators to improve communica- 
tion and commonality of concept. 

Regarding SUNY's interface with external agen- 
cies it was recommended that: 



• Central Administration and the Division of the Budget 
jointly plan and establish annual policy studies of systemwide 
concern, to be conducted by either, partly or jointly, where 
appropriate. 

• The State Education Department and Central Adminis- 
tration develop a procedure for the conduct of doctoral program 
evaluations and on-site consultant visitations for which SUNY 
would have the primary responsibility. 

• The fiscal and program committees of the legislature 
conduct interim studies and gather information on SUNY issues 
of statewide significance such as the utilization and cost ef- 
fectiveness of educational technology, and the adequacy of 
statewide student financial aid programs. 



In the area of planning, the study suggested the 
following recommendations: 

• That campus master plans , rovlde more delineated state- 
ments of mission m terms of specific programs and objectives 
upon which the campus intends to focus its efforts and con- 
centrate its resources. 

• That campuses implement systems of internal goal setting 
and self-evaluation for planning purposes. 

• That Central Administration establish a performance audit- 
ing staff to conduct reviews of plan Implementation and provide 
technical assistance to the campuses for planning and base 
budget review. ^ 
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To improve Central's capacity to measure campus 
performance, provide maximum campus flexibility 
and establish commonly understood criteria for pro- 
gram review and post audit, it was recommended 
that: 

• SUNY develop a procedure for the submission of a pro- 
grammatic budget justification narrative which includes state- 
ments of goals and objectives, descriptions of prior accom- 
plishments and informal criteria for the evaluation of goal 
achievement. 

• There be further development of a NCHEMS* based cost 
per program methodology and the setting of ranges and stand- 
ards for SUNY program costs for purposes of program planning 
and internal resource allocation. 



ACCOUNTABILITY EVALUATION 
EFFECTIVENESS, QUALITY 



EFFICIENCY. 



In addition to clarification of accountability roles, 
a system of accountability requires clarification of 
measuring performance. In higher education perfor- 
mance is judged in terms of efficiency, effectiveness 
and quality. Each of these concepts is difficult to 
measure, and indeed they are often addressed on the 
basis of proxy measures or intuitive criteria. A se- 
cond difficulty is that efficiency is often viewed as 
incompatible with a commitment to quality. Finally, 
effectiveness is often confused with efficiency by 
individuals for whom efficiency is a primary criteria, 
or confused with quality by those who reject the 
legitimacy of efficiency as it applies to university 
programs. 

These ambiguities stem from the relative priority 
given to each by the various levels of university gov- 
ernance. For example, the New York state external 
authorities seem committed to efficiency considera- 
tions as they express the need for more objective 
measures of university performance based on cost 
factors. SUNY respondents, on the other hand, 
tended to emphasize considerations of effective- 
ness and quality. Indeed, some university respon- 
dents argued that efficiency criteria are not appli- 
cable to university activities and they preferred the 
concept of effectiveness. In addition, there was dis- 
agreement as to whether specific items used in the 
study's questionnaire were efficiency or effective- 
ness or quality criteria. 

This confusion over meanings and relative prior- 
ities i3 another illustration of the competing concep- 
tions of what a public university should be. In New 
York, as in other states, there is a multiplicity of lan- 
guages among participants in the governance pro- 



*NCHEMS is the National Center for Higher Education Manage- 
ment Systems that has developed an educational cost analysis 
system (lEP) which allows colleges and universities to identify 
direct and indirect Instructional costs by academic program 
using standard methodologies and accounting procedures. 
SUNY is completing a pilot implementation of a NCHEMS-based 
cost analysis system. 



cess. There are many assumptions based on con- 
ventional wisdom about the relationships of effic- 
iency, effectiveness and quality, such as the notion 
that quality is generally associated with high cost, 
or that high levels of investment will somehow di- 
rectly result in quality. This appears to be the case 
in New York, as well as nationally. 

Our study attempted to analyse the meanings of 
these concepts, chart their interrelationships, and 
discuss problems in their application. The objective 
was to set the tone for the emergence of a consen- 
sus by SUNY and state governmental reviewing 
authorities on the meaning of terms, and encourage 
the development of a set of working definitions for 
efficiency, effectiveness and quality. 

As an applied term, efficiency can be defined as 
the relationship of cost to units of production. But 
both cost and production present difficulties. The 
units of production are preferably units of output, 
but there is currently no adequate consensus on 
educational outputs, much less on means of mea- 
suring these outputs. In practice, proxy measures 
are employed which tend to identify units of input 
such as student/faculty ratios, cost per student, 
cost per faculty member, cost per square foot of 
useable space and so on. 

Some respondents to our study criticized effic- 
iency as requiring too narrow a focus on costs. 
Since higher education is labor intensive, savings 
in cost must be achieved by savings in labor which 
is generally assumed to have a detrimental effect on 
quality. Ideally, however, cost or efficiency should 
be only one of several criteria for decision making. 
In this view, cost data are neutral and inform choice 
rather than supplant human judgment. 

This broader understanding of choice or judgment 
requires an understanding of effectiveness. Effec- 
tiveness can be measured by the degree to which an 
organization or program achieves its goals i, and 
must be determined by specified criteria or estab- 
lished standards of some kinds^. Thus, effective- 
ness of an educational activity cannot be determined 
exclusively on the basis of cost but requires a set 
of objectives or goals, or more broadly, a sense of 
purpose. 

There is a meeting ground between efficiency and 
effectiveness such as the notion of cost effective- 
ness which blends both considerations. Cost con- 
sciousness goes beyond budget consciousness. It 
considers how hired faculty and enrolled students 
interact for some purpose^. Cost effectiveness re- 
lates to the setting of priorities among competing 
choices for which both cost (efficiency) and the 
compatability of program design with objectives 
and broader societal needs (effectiveness) serve 
as criteria. 

Quality in higher education is more difficult to 
define than it is to recognize. That is to say, no one 
would deny its presence or absence in various sit- 
uations even though we might disagree on how to 
measure it. Whereas effectiveness can be identified 



in terms of goal achievement and the corresponding 
criteria for goal achievement can be objectively 
specified in the initial goal description, quality is a 
very relative and ultimately subjective attribute. 
There are great ambiguities and subtleties surround- 
ing it. 

In practice our study indicated that both respon- 
dents from the SUNY system and those from exter- 
nal agencies often used quality and effectiveness 
interchangeably. Judgments of effectiveness often 
implied judgments about academic quality since 
effectiveness was often viewed as achieving certain 
quality standards. Similarly, formulations of quality 
often paralleled the criteria used for the evaluation 
of effectiveness. Conceivably, however, a quality 
program as defined by traditional standards of aca- 
demic excellence and measured by scholarly repu- 
tation of faculty and significance and volumeof pub- 
lished research, may not be deemed to be effective 
in terms of economies of scale, complementarity 
with other university programs, efficiency as mea- 
sured by the cost per completed degree, or respon- 
siveness to a changing market for Ph.D.'s. On the 
other hand, an educational program may be deemed 
to be effective in meeting certain limited objectives, 
without achieving other absolute standards of aca- 
demic quality. 

The relationship between quality and efficiency Is 
also, our study indicated, interdependent. Strictly 
speaking quality cannot be measured, but it can be 
compared^. It is widely assumed that there is a di- 
rect relationship between quality and cost, or con- 
versely that quality decreases as efficiency in- 
creases. Upon closer analysis, this relationship is 
far more complex and the evidence is mixed. 

One indicator which is used for budgeting pur- 
poses and for quality comparisons in New York is 
the stuaent/faculty ratio. A national study of grad- 
uate programs indicated that there is a strong posi- 
tive relationship between faculty/student ratios and 
program quality as measured by Cartter's study of 
quality in graduate education and the numbers of 
Woodrow Wilson fellows entering each institution. 
This relationship was stronger in private universities 
than for public universities, however^. Studies have 
also revealed a wide variation in faculty/student 
ratios among similar types of Institutions, and a- 
mong institutions of similar quality^.This is due to 
the fact that existing ratios are in large part a result 
of unplanned historical evolution and the result of 
an internal competition for resources and the availa- 
bility of outside resources to various departments 
and programs. Consequently, the Carnegie Com- 
mission staff concluded that it is unlikely that over- 
all faculty/student ratio in any given institution is 
optimal ^. This also indicates that the relationship of 
student/faculty ratios to program quality is not di- 
rect since wide variations exist within the same lea- 
gues of quality programs. There are, however, ap- 
parent relationships between program size, program 
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quality and efficiency. |n general, the graduate pro- 
grams that are rated high on the Cartter rankings of 
quality tend to be the larger programs which, in turn, 
are also more efficient due to economies of scaled. 

Even this relationship can be challenged, how- 
ever. Powel and Lamson performed regression anal- 
yses on data drawn from graduate programs nation- 
ally to test the effects of program size and program 
quality (as measured by national ratings) on unit 
costs. They reported mild indications of some econ- 
omies of scale (larger programs had lower unit 
costs), but there were also some indications that 
higher quality programs tended to have higher unit 
costs (as measured by direct instructional costs 
per FTE graduate student)^. 

Program costs vary widely, quite independent of 
quality standards or type of institution. The Powe) 
and Lamson study of masters and doctoral pro- 
grams showed that on a discipline-by-discipline 
basis the range in costs from the lowest cost pro- 
gram to the highest was often a difference of 300 to 
400 percent^o Another study by McKinsey and Com- 
pany revealed wide variations in costs per tudent 
among institutions of similar quality^i. Th would 
indicate that there are wide differences ii osts 
(as well as faculty /student ratios) which c »t be 
explained solely by differences in quality. 

A working formula of evaluation can be suggested 
that views efficiency, effectiveness and quality as 
interdependent variables: 

Effectiveness = (efficiency) plus quality standards ""2 . 



Consequently, the following checklist^^categorizes 
the criteria and indicators in terms of the three var- 
iables of input, process and output. 

There has also been a tendency to blur the distinc- 
tion between general criteria and the specific indica- 
tors designed to inform those criteria. For that rea- 
son the checklist is an attempt to differentiate where 
possible between criteria and indicators. 

The checklist can also provide a guide for review 
of SUNY undergraduate programs as well as grad- 
uate programs, with the following exceptions. There 
tends to be a higher degree of mutual dependence 
among undergraduate programs. Consequently, the 
criteria of program compatibility and interdepen- 
dence are probably more critical at the undergrad- 
uate levelthanat the graduate level. Second, criteria 
such as research productivity of faculty and stu- 
dents and responsiveness of programs to specific 
employment conditions are more relevant to grad- 
uate than to undergraduate programs. 

The program review criteria and indicators are 
offered as a basis for the SUNY governance system 
to work toward consensus in its method of evaluat- 
ing efficiency, effectiveness and quality. By delin- 
eating and clarifying the evaluation process, the 
checklist can help to minimize confusion and en- 
large the areas of agreement in operating a multi- 
campus system of accountability. 

Leif Hartmark 
Project Coordinator 



Under this formulation cost is seen as an input 
variable, effectiveness is measured in terms of out- 
puts, and quality standards are viewed as an inter- 
vening variable. The effectiveness of New York 
higher education can then be defined as the condi- 
tion in which outputs are achieved efficiently, that 
is. within a tolerable limit of unit costs con- 
forming to predetermined goals that include 
standards of quality. 



Checklist of Criteria 

As the above indicates there has been a failure in 
measuring performance to distinguish and relate 
input variables, process variables, and output vari- 
ables. Many previous evaluations by accrediting 
agencies and external reviewers have tended to fo- 
cus on the input variables of quality and reputation 
of faculty, quality of students, and adequacy of re- 
sources. This orientation overlooks the impact of a 
program upon the ultimate success of its students 
and, of equal importance, it tends to neglect the 
manner in which teachers and learners interact for 
some purpose. For example, rather than focusing 
on the adequacy of library resources, the concern 
should be with how well these resources are ibed. 
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CHECKLIST OF SELECTED CRITERIA AND INDICATORS FOR 
QUALITY ASSESSMENT AND PROGRAM REVIEW 



Variable 



Criterion 



Indicator 



A. Input 

1. Faculty Quality 



Reputation 



-Faculty membership in professional associations, honorary 
societies, editorial boards, other professional leadership pos- 
itions 

-Faculty participation in peer review of nationally competitive 
grant proposals 
-Outside consultancies 

-Evaluation of faculty by outside consulting agencies (federal, 
state, regional) 
-Faculty honors and awards 

-Peer evaluations 
-Reputational analysis 
-Anderson-Cartter graduate ratings 



2. Students Quality 



-Entering student scores on standardized tests 
-Grade point averages 

-Rank in class, former educational institution 
-National fellowships held 
-Trends in entrance standards 

-Faculty assessment of students' probability of successful de- 
gree completion 



3. Financial Adequacy of Financial 
Resources Support 



-Total program budget (state and non-state funds) 
-Ratio of non-state to state funds 
-Median and range in faculty salaries 
-Competitive position of salaries compared to departments 
in comparable institutions 

-Number of "stars" who were recruited away compared to num- 
ber of "stars" brought in from outside 
-Organized research or public service centers or institutes 
-Average competitive extramural funds attracted per faculty 
member 

-Level of financial support for students 



4. Physical Adequacy of Physical 
Resources Plant 



-Net assignable sauare feet per faculty member 
-Relative isolation/integration of facility from university pro- 
grams and services 

-Adequacy of study space, contact offices, laboratory space 
and equipment, computing facilities, etc. 



5. Supporting Adequacy of Library 
Services 



Computing Facilities 



-Total number of holdings in field 
-Rate of acquisitions 
-Access to materials 

-Interlibrary loan and computerized networks 

-Access to computing time 
-Cost of computing time 

-Efficiency in terms of extent of utilization and competitive- 
ness of cost 

-Quality and capacity of hardware 
-Versatility of programming capability 



Media Services and 
Educational Technology 

Department Support 



-Extent of utilization 

-Extent of media assisted curriculum development 

-Clerical personnel per FTE faculty, per FTE student 
-Access to and adequacy of departmental services 



t 



EMC 



Supporting Disciplines 



-Existence of quality programs in related disciplines 

6 



Variable 



CHECKLIST [continued] 
Criterion Indicator 



B. Process 



Faculty Wortcload 



-Student access to senior professors 
-Faculty teaching load 
-Faculty advisement load 



Efficiency 



-Cost per student year 
-Cost per completed degree 

-Average length of time required for completion of degree 
-Number of graduates per year 
-Total costs to the student per degree 



Pedagogical Methods 
and Content 



-Extent of instructional innovation 

-Degree to which course content and research reflects the cut- 
ting edge of the field 



Program Interaction 



-Reciprocal contributions to and from related disciplines and 
programs on campus 

-Data on student flow among programs 

-Extent of interdisciplinary instruction and research 



Morale 



-Student morale (interviews) 
-Faculty morale (interviews) 



C. Output 



Faculty Research 



-Publication count, citation indices 

-Caliber of publications in which faculty research appears 
-Judgments of importance of the research by peers, state and 
national policy-study commissions or other external agencies 



Student Research 



-Student participation in faculty research 
-Student research publications 

-Quality of dissertations as judged by external peer reviews of 
dissertations in the field 



Success of Graduates 



-Graduate placement 

-Starting salaries of graduates of professional/occupational 
programs 

-Surveys of employer satisfaction with graduates 
-Graduate's career records after 5, 10 years 



Educational Outcomes 



-Student value added (e.g. ratios of student test scores on 
achi>vement tests upon entrance and graduation) 

-Surveys of expectations of entering students compared to 
surveys of student satisfaction upon graduation 



Program Vitality 



-Program enrollment trends compared to changes in national 
and regional manpower needs 

-Success of graduates in adapting to alternate career oppor- 
tunities for Ph.D's in government and industry 
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NOTES Icont.] 



9. See J. H. Powel and R. D. Lamson. El«m«nts R«lat«d to th« 
D«t«nninatSoi of Costs and Bansflts of Graduata Education, 

Washington, DC: Council of Graduate Schools, 1972; J. L. 
McCarthy and D. R. Deener, Tha Costs and Banaflts of Qrad- 
uata Education: Commantary with Racommandations, 

Washington, DC: Council of Graduate Schools, 1972. For a 
discussion of these and similar studies and a presentation of 
the Powel and Lamson cost data on graduate programs see 
Frederick E. Balderston, IManaging Today's Univarsity (San 
Francisco: Jossey-Bass Publishers, 1974) pp. 164-77. 



10. Balderston, op. cit., p. 175. 



11. McKinsey and Company, Tha Twaiva Coilaga Jst-Quality 
Study Washington, DC, 1972, cited in Carnegie Commis- 
sion, op. cit., p. 45. 



12. This formulation is based on a definition of productivity pre- 
sented by Waiter Balk in "Fundamental Concepts and the 
Measurement of Productivity," Admlnlstaring Stata Govam- 
mani Productivity improvamant Programs, Ford Foundation 
Productivity Research Project for State Government, Albany, 
New York, September 1974. 

13. The checklist is based upon our research and an analysis of 
several approaches to program review including the New York 
State Regents doctoral review; the SUNY guidelines for 
graduate program review; the University of California-Berke- 
ley doctoral program review; the Princeton University Prior- 
ities Committee; and checklists suggested by Frederick Bal- 
derston. For the basic approach and guidelines of the Re- 
gents, SUNY, and the University of California at Berkeley 
see Regents Commission on Doctoral Education, op. cit. 
pp. 30, 65-72. For a discussion of the Princeton criteria and 
additional considerations for program review see Fred- 
erick E. Balderston, op. cit., pp. 264-69, 283-87. 
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